<HTML>
<HEAD>
<!-- This HTML file has been created by texi2html 1.29
     from lex.tnf on 19 December 2010 -->

<TITLE>Lexical Analysis - Case Insensitivity</TITLE>
</HEAD>
<BODY TEXT="#000000" BGCOLOR="#FFFFFF" LINK="#0000EE" VLINK="#551A8B" ALINK="#FF0000" BACKGROUND="gifs/bg.gif">
<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0" VALIGN=BOTTOM>
<TR VALIGN=BOTTOM>
<TD WIDTH="160" VALIGN=BOTTOM>
<A HREF="http://eli-project.sourceforge.net/">
<IMG SRC="gifs/elilogo.gif" BORDER=0>
</A>&nbsp;
</TD>
<TD WIDTH="25" VALIGN=BOTTOM>
<img src="gifs/empty.gif" WIDTH=25 HEIGHT=25>
</TD>
<TD ALIGN=LEFT WIDTH="475" VALIGN=BOTTOM>
<A HREF="index.html"><IMG SRC="gifs/title.png" BORDER=0></A>
</TD>
<!-- |DELETE FOR SOURCEFORGE LOGO|
<TD>
<a href="http://sourceforge.net/projects/eli-project">
<img
  src="http://sflogo.sourceforge.net/sflogo.php?group_id=70447&amp;type=13"
  width="120" height="30"
  alt="Get Eli: Translator Construction Made Easy at SourceForge.net.
    Fast, secure and Free Open Source software downloads"/>
</a>
</TD>
|DELETE FOR SOURCEFORGE LOGO| -->
</TR>
</TABLE>

<HR size=1 noshade width=785 align=left>
<TABLE BORDER=0 CELLSPACING=2 CELLPADDING=0>
<TR>
<TD VALIGN=TOP WIDTH="160">
<h4>General Information</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="index.html">Eli: Translator Construction Made Easy</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="gindex_1.html#SEC1">Global Index</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="faq_toc.html" >Frequently Asked Questions</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Tutorials</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="EliRefCard_toc.html">Quick Reference Card</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="novice_toc.html">Guide For new Eli Users</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="news_toc.html">Release Notes of Eli</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="nametutorial_toc.html">Tutorial on Name Analysis</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="typetutorial_toc.html">Tutorial on Type Analysis</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Reference Manuals</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ui_toc.html">User Interface</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="pp_toc.html">Eli products and parameters</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lidoref_toc.html">LIDO Reference Manual</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Libraries</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lib_toc.html">Eli library routines</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="modlib_toc.html">Specification Module Library</a></td></tr>
</table>

<h4>Translation Tasks</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lex_toc.html">Lexical analysis specification</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="syntax_toc.html">Syntactic Analysis Manual</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="comptrees_toc.html">Computation in Trees</a></td></tr>
</table>

<h4>Tools</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lcl_toc.html">LIGA Control Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="show_toc.html">Debugging Information for LIDO</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="gorto_toc.html">Graphical ORder TOol</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="fw_toc.html">FunnelWeb User's Manual</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ptg_toc.html">Pattern-based Text Generator</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="deftbl_toc.html">Property Definition Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="oil_toc.html">Operator Identification Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="tp_toc.html">Tree Grammar Specification Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="clp_toc.html">Command Line Processing</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="cola_toc.html">COLA Options Reference Manual</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="idem_toc.html">Generating Unparsing Code</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="mon_toc.html">Monitoring a Processor's Execution</a> </td></tr>
</table>

<h4>Administration</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="sysadmin_toc.html">System Administration Guide</a> </td></tr>
</table>

<HR WIDTH="100%">
<A HREF="mailto:eli-project-users@lists.sourceforge.net">
<IMG SRC="gifs/button_mail.gif" BORDER=0 ALIGN="left"></A>
<A HREF="index.html"><IMG SRC="gifs/home.gif" BORDER=0 ALIGN="right"></A>

</TD>
<TD VALIGN=TOP WIDTH="25"><img src="gifs/empty.gif" WIDTH=25 HEIGHT=25></TD>

<TD VALIGN=TOP WIDTH="600">
<H1>Lexical Analysis</H1>
<P>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_4.html"><IMG SRC="gifs/prev.gif" ALT="Previous Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_6.html"><IMG SRC="gifs/next.gif" ALT="Next Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_toc.html"><IMG SRC="gifs/up.gif" ALT="Table of Contents" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT="">
<HR size=1 noshade width=600 align=left>
<H1><A NAME="SEC23" HREF="lex_toc.html#SEC23">Case Insensitivity</A></H1>
<P>
The default behavior of an Eli-generated lexical analyzer is to treat each
ASCII character as an entity distinct from all other ASCII characters.
This behavior is inappropriate for applications that do not distinguish
upper-case letters from lower-case letters in certain contexts.
For example, a Pascal compiler ignores the case of letters in identifiers
and keywords, but distinguishes them in strings.
Thus the Pascal identifiers <CODE>MyId</CODE>, <CODE>MYID</CODE> and <CODE>myid</CODE> are
identical but the strings <CODE>'MyString'</CODE>, <CODE>'MYSTRING'</CODE> and
<CODE>'mystring'</CODE> are different.
<P>
Case insensitivity is reflected in the identity of character sequences.
In other words, the character sequences <CODE>MyId</CODE>, <CODE>MYID</CODE> and
<CODE>myid</CODE> are considered to be identical character sequences if and only
if the generated processor is insensitive to the case of letters.
Two character sequences are identical as far as the remainder of the
processor is concerned if they have the same classification and their
values are equal (see  <A HREF="lex_1.html#SEC1">Specifications</A>).
Since the classification and value are determined by the token processor,
it is the token processor that must implement case insensitivity.
<P>
Two conditions must be met if a processor is to be insensitive to case:
<P>
<OL>
<LI>
A token processor that maintains a table of character sequences
in which all letters are of one case must be available.
<P>
<LI>
The specification of each case-insensitive character sequence
must invoke such a token processor.
</OL>
<P>
<H2><A NAME="SEC24" HREF="lex_toc.html#SEC24">A Case-Insensitive Token Processor</A></H2>
<P>
The token processor <CODE>mkidn</CODE>
<A NAME="IDX158"></A>
maintains a table of character sequences
and provides the same classification and value for
identical character sequences.
Normally, <CODE>mkidn</CODE> treats upper-case letters and lower-case letters as
different characters.
This behavior is controlled by an exported variable, <CODE>dofold</CODE>
(see  <A HREF="lib_1.html#SEC8">Unique Identifier Management of Library Reference Manual</A>):
When <CODE>dofold=0</CODE> character sequences are entered into the table as they
are specified to <CODE>mkidn</CODE>; otherwise all letters in the sequence are
converted to upper case before the sequence is entered into the table.
<P>
Although the value of <CODE>dofold</CODE> could be altered on the basis of
context by user-defined code, it is normally constant throughout the
processor's execution.
To generate a processor in which <CODE>dofold=1</CODE>, specify the parameter
<CODE>+fold</CODE> in the request
(see  <A HREF="pp_1.html#SEC5">fold -- Make the Processor Case-Insensitive of Products and Parameters Reference Manual</A>).
If this parameter is not specified in the request, Eli will produce a
processor with <CODE>dofold=0</CODE>.
<P>
The value set by <CODE>mkidn</CODE> is the (unique) index of
the transformed character sequence in the table.
Thus if that value is used to retrieve the sequence at a later time, the
result will be the original sequence with all lower-case letters replaced
by their upper-case equivalents.
<P>
<H2><A NAME="SEC25" HREF="lex_toc.html#SEC25">Making Literal Symbols Case Insensitive</A></H2>
<P>
Since literal symbols are recognized exactly as they stand in the grammar,
they are case sensitive by definition.
For example, if a grammar for Pascal contains the literal symbol
<CODE>'begin'</CODE> then the generated processor will recognize only the
character sequence <CODE>begin</CODE> as an instance of that literal symbol.
This behavior could be changed by redefining the literal symbol as a
nonliteral symbol (say) <CODE>BEGIN</CODE>, and providing the following
specification in a type-<TT>`gla'</TT> file:
<P>
<PRE>
BEGIN:  $[Bb][Ee][Gg][Ii][Nn]  [mkidn]
</PRE>
<P>
If the number of literal symbols to be treated as case-insensitive is
large, this is a very tedious and error-prone approach.
It also distorts the grammar by converting literal terminal symbols
to non-literal terminal symbols.
<P>
To solve this problem, Eli allows the user to specify a set of literal
symbols that should be placed into the table used by <CODE>mkidn</CODE>, with
their classification codes, at the time the generated lexical analyzer is
loaded.
If the <CODE>+fold</CODE> parameter is also specified, all lower-case letters in
these symbols will be replaced by their upper-case equivalents before the
symbol is placed into the table.
The desired behavior is then obtained by invoking <CODE>mkidn</CODE> after
recognizing the appropriate character sequence in the input text.
<P>
The set of literal symbols to be placed into the table is specified by
giving a sequence of regular expressions in a type-<TT>`gla'</TT> file, and
then deriving the <CODE>:kwd</CODE> product from that file
(see  <A HREF="pp_2.html#SEC15">kwd -- Recognize Specified Literals as Identifiers of Products and Parameters Reference Manual</A>).
The regular expressions describe the form of the literal symbols in the
grammar, <EM>not</EM> the input character sequences to be recognized.
<P>
Suppose, for example, that a Pascal grammar specified all keywords as
literal symbols made up of lower-case letters:
<P>
<PRE>
Statement:
  ...
  'while' Expression 'do' Statement /
  ...
</PRE>
<P>
A type-<TT>`gla'</TT> file describing the form these symbols take in the
grammar would consist of the single line <CODE>$[a-z]+</CODE>.
If the name of that file was <TT>`PascalKey.gla'</TT> then the user could tell
Eli to initialize <CODE>mkidn</CODE>'s table with all of the keywords by
including the following line in a type-<TT>`specs'</TT> file:
<P>
<PRE>
PascalKey.gla :kwd
</PRE>
<P>
In Pascal, keywords have the form of identifiers in the input text.
Therefore the canned description <CODE>PASCAL_IDENTIFIER</CODE> suffices to
recognize both identifiers and keywords.
<CODE>PASCAL_IDENTIFIER</CODE> invokes <CODE>mkidn</CODE> to obtain the classification
and value of the sequence recognized by the regular
expression <CODE>$[a-zA-Z][a-zA-Z0-9]*</CODE>.
Since <CODE>mkidn</CODE>'s table has been initialized with the character
sequences for the literal keyword symbols, and their classifications,
they will be appropriately recognized.
<P>
The <CODE>:kwd</CODE> product and the <CODE>+fold</CODE> parameter are independent of
one another.
Thus, in order to make the generated lexical analyzer accept Pascal
keywords with arbitrary case the user must both provide the <CODE>:kwd</CODE>
specification and derive with the <CODE>+fold</CODE> parameter.
<P>
<HR size=1 noshade width=600 align=left>
<P>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_4.html"><IMG SRC="gifs/prev.gif" ALT="Previous Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_6.html"><IMG SRC="gifs/next.gif" ALT="Next Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_toc.html"><IMG SRC="gifs/up.gif" ALT="Table of Contents" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT="">
<HR size=1 noshade width=600 align=left>
</TD>
</TR>
</TABLE>

</BODY></HTML>
